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Abstract 

Background: Diminishing stigmatization for those with mental illnesses by health care providers (HCPs) is 
becoming a priority for programming and policy, as well as research. In order to be successful, we must accurately 
measure stigmatizing attitudes and behaviours among HCPs. The Opening Minds Stigma Scale for Health Care 
Providers (OMS-HC) was developed to measure stigma in HCP populations. In this study we revisit the factor 
structure and the responsiveness of the OMS-HC in a larger, more representative sample of HCPs that are more 
likely to be targets for anti-stigma interventions. 

Methods: Baseline data were collected from HCPs (n = 1,523) during 12 different anti-stigma interventions across 
Canada. The majority of HCPs were women (77.4%) and were either physicians (MDs) (41.5%), nurses (17.0%), 
medical students (13.4%), or students in allied health programs (14.0%). Exploratory factor analysis (EFA) was 
conducted using complete pre-test (n = 1,305) survey data and responsiveness to change analyses was examined 
with pre and post matched data (n = 803). The internal consistency of the OMS-HC scale and subscales was 
evaluated using the Cronbach's alpha coefficient. The scale's sensitivity to change was examined using paired 
t-tests, effect sizes (Cohen's d), and standardized response means (SRM). 

Results: The EFA favored a 3-factor structure which accounted for 45.3% of the variance using 1 5 of 20 items. The 
overall internal consistency for the 15-item scale (a = 0.79) and three subscales (a = 0.67 to 0.68) was acceptable. 
Subgroup analysis showed the internal consistency was satisfactory across HCP groups including physicians and 
nurses (a = 0.66 to 0.78). Evidence for the scale's responsiveness to change occurred across multiple samples, 
including student-targeted interventions and workshops for practicing HCPs. The Social Distance subscale had the 
weakest level of responsiveness (SRM < 0.50) whereas the more attitudinal-based items comprising the Attitude 
(SRM < 0.91) and Disclosure and Help-seeking (SRM < 0.68) subscales had stronger responsiveness. 

Conclusions: The OMS-HC has shown to have acceptable internal consistency and has been successful in detecting 
positive changes in various anti-stigma interventions. Our results support the use of a 15-item scale, with the 
calculation of three sub scores for Attitude, Disclosure and Help-seeking, and Social Distance. 

Keywords: Stigma, Scales, Measurement instruments, Mental-health-related stigma, Attitudes, Contact, Intervention, 
Psychometrics 
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Background 

Stigma can be defined as 'the co-occurrence of labeling, 
stereotyping, separation, status loss, and discrimination 
in a situation where power is exercised" [1]. This defin- 
ition speaks to the complexity of the stigma construct. 
Similarly, the research literature has identified various 
aspects related to and sub-components of the stigma 
construct, including perceived stigma [2-5], self-stigma 
[6], social distance [3,7], danger/violence [8], helping [8], 
negativism, as opposed to a belief in recovery [9], and 
emotional reactions [10] including social responsibility 
and lack of empathy or comparison towards people with 
mental illness. Corrigan and colleagues suggest that 
"stigma related to mental illness represents a significant 
public health concern because it is a major barrier to 
care seeking or ongoing treatment participation" [11]. The 
negative effects of stigmatization can have a substantial 
negative impact on patient quality of life [12,13]. In 
addition, stigmatizing attitudes or behaviours by health 
care providers have the potential to lead to a lack of access 
to care, to under- treatment [14], to social marginalization 
[15], and can undermine the relationship between the pa- 
tient and provider. Perhaps most importantly, stigmatizing 
attitudes by health care providers are likely to compound 
a person's feelings of rejection and isolation and be a real 
barrier to receiving appropriate care [11]. For these rea- 
sons, diminishing stigmatization by health care providers 
is increasingly becoming a priority for both programming 
and policy, as well as research. In order to be successful, 
studies must be able to systematically and accurately meas- 
ure stigmatizing attitudes among health care providers. 
The Opening Minds Scale for Health Care Providers 
(OMS-HC) [16] was developed to evaluate anti-stigma 
interventions in health care provider populations. 

A research team, which consisted of a sociologist, 
psychiatrist, anti-stigma program manager, and research 
associate, began the development of the OMS-HC by 
generating an item-pool from new and existing scales. 
After item-pool generation and content validation via 
cognitive interviews, focus groups were held with 64 
health care providers/trainees and six people with lived 
experience of mental illness to determine if the language 
and tone were appropriate and that no concepts were 
missing. After these focus groups, an initial 20-item ver- 
sion of the OMS-HC was created and administered to 
787 health care providers/trainees across Canada. In the 
testing phase, one third of respondents were nurses/ 
nursing students (17.5%) or physicians/medical students 
(16.4%). A higher proportion of respondents were either 
social workers (21.2%), psychiatric nurses or psychologists 
(25.8%), or other types of health care providers such 
as pharmacists, recreational therapists, and counselors 
(19.1%). A factor analysis of the scale data from this sam- 
ple resulted in a two-factor solution (i.e., attitudes factor 



and helping/disclosure factor) with 12 items. Interestingly, 
an anticipated social distance factor did not materialize 
from the factor analysis. In earlier phases of development 
of the scale [16], consultations with international experts 
in the field of stigma related to mental health had indi- 
cated that social distance was an important part of the 
stigma construct. Subsequently, item selection, partially 
guided by construct validation by experts, had included 
items related to social distance. 

As we have accumulated extensive data from real-world 
applications, we sought to revisit the factor structure 
and the responsiveness of the OMS-HC in a larger more 
representative sample of health care providers that are 
more likely to be targets for anti-stigma interventions. 
The reason for this is threefold. The 12-item version 
appeared to miss the important dimension of social dis- 
tance; this is an important dimension to capture, as it 
can be considered a proxy for behavioural discrimin- 
ation and behavioural intentions (i.e., measuring a will- 
ingness to readily engage persons with mental illness in 
various activities and relationships) [17]. Second, the 
make-up of respondent pool for the original scale test- 
ing was less than ideal in that it under-represented phy- 
sicians and nurses and more heavily represented social 
workers, psychiatric nurses, and psychologists. Finally, 
the availability of a large pool of data will allow us to de- 
termine the responsiveness of the OMS-HC in measur- 
ing stigmatizing attitudes among a variety of health care 
providers. 

Methods 

Research design 

This study is a secondary analysis of data collected for a 
large evaluative study examining the impact of various 
mental health anti-stigma interventions conducted in part- 
nership with the Mental Health Commission of Canada 
through its Opening Minds initiative [18,19]. 

Study setting 

Mental Health Commission of Canada: Opening Minds 

"As part of its 10-year mandate, the Mental Health 
Commission of Canada (MHCC) has embarked on an 
anti-stigma initiative called Opening Minds (OM) to 
change the attitudes and behaviours of Canadians towards 
people with a mental disorder. A key component of pro- 
grams being evaluated is contact-based educational ses- 
sions, where target audiences hear personal stories from 
and interact with individuals who have experience with 
mental disorders and have recovered or are managing 
their illness. OM's goal is to evaluate programs nationally, 
develop new interventions to address gaps in existing 
programs, and add other target groups over time. OM 
and the MHCC are funded by a grant from Health 
Canada" [18]. 
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Sample 

For this study, baseline data was collected from partici- 
pants (n = 1,523) from 12 different anti-stigma programs 
that employed the full 20-item OMS-HC as a component 
of the program's evaluation by the MHCC. Additional 
file 1 describes each intervention and its corresponding 
target audience. The OMS-HC was administered to each 
audience prior to the anti-stigma intervention being 
received with a post-test administered afterwards. In all 
but a few of the sites, baseline demographic information 
was also collected, including gender, age, profession, 
and medical specialization. Testing of psychometric 
properties was completed for respondents who fully 
completed the OMS-HC survey and matched analysis 
was undertaken for respondents with complete and 
matched pre- and post-test surveys. In line with the rec- 
ommendation for heterogeneous samples in factor analysis 
[20], participants represented diverse types of health care 
providers with varying ages and at varying stages of their 
careers, including trainees. Access to the OMS-HC ratings 
occurred under data sharing agreements between the anti- 
stigma programs and the MHCC. Our study was approved 
by the Conjoint Health Research Ethics Board at the 
University of Calgary. 

The Opening Minds Scale for Health Care Providers 
(OMS-HC) 

The Opening Minds Scale for Health Care Providers 
(OMS-HC) is a self- report questionnaire, which assesses 
attitudes and behavioural intentions towards people with 
mental illness. The scale consists of a series of items, each 
having the following balanced response values: strongly 
agree, agree, neither agree nor disagree, disagree, strongly 
disagree, such that each item is assigned a score of 1 to 5. 
High scores suggest a more stigmatizing attitude. The full 
OMS-HC contains 20 items with scores that can range 
from 20 (least stigmatizing) to 100 (most stigmatizing). 
Items 3, 8, 9, 10, 11, 15, 19 require reverse coding. 

Statistical analysis 

Sample characteristics were examined using descriptive 
statistics. Mean subscale scores were calculated by sum- 
ming the raw scores for items within each factor divided 
by the number of items. The internal consistency of the 
OMS-HC scale and subscales was evaluated using the 
Cronbach's alpha coefficient and construct validity for 
the various subscales was examined using the Pearson 
correlation (r). Data were analyzed using STATA 12.0 [21]. 

EFA of the OMS-HC to assess the factor structure 

Exploratory factor analysis was carried out to determine 
the construct validity of the OMS-HC. Factors were ex- 
tracted using eigenvalue-one procedure, Catell's scree 
test [22], and Horn's parallel analysis [23-25]. A principal 



component analysis (PCA) with orthogonal varimax pro- 
cedure was employed to rotate the factors to a simple 
structure in order to determine the number of factors to 
retain. Items with a loading of greater than 0.4 were 
retained for a specific factor [26] . Split items were retained 
to factors if the square of the loadings for a factor 
was >50% that of its loading on any other factor [26]. 
Prior to EFA, Bartlett's Test of Spehericity and Kaiser- 
Meyer-Olkin (KMO) measure of sampling adequacy were 
calculated to verify the appropriateness of using factor 
models [27]. The KMO shows that the sample size is 
adequate as the ratio of the sample size to the number of 
items is sufficient for factorability [27]. When the Bartlett's 
Test of Spehericity is statistically significant, it supports 
the factorability of the correlation matrix [27]. These two 
measures of sampling adequacy imply the suitability of 
applying the factor analysis to the available data. 

Responsiveness to change 

A scale's responsiveness is defined as its ability to detect 
important change [28] and its ability to express the size 
of the effect of an intervention, assuming that most of 
the interventions are having positive effects. The scale's 
sensitivity to change was examined in three ways: (1) a 
paired t-test, which tested if there was a difference be- 
tween pre- and post-intervention scores [29]; (2) an effect 
size (Cohen's d), which calculates the extent of change in a 
standardized form, thereby allowing for comparison be- 
tween the questionnaires [30]; and (3) the standardized re- 
sponse mean (SRM), which differs from the effect size in 
that it takes into account the variability in change instead 
of the baseline scores [31]. Although there is some debate 
in the literature about the use of benchmarks for assessing 
the relative size of change in the SRM, effect sizes can be 
(and often are) translated into benchmarks; an effect size 
of 0.2 is considered small, 0.5 is considered medium, and 
0.8 or greater is considered a large effect [30,32]. In order 
to determine if the OMS-HC was sensitive to detect 
changes in the different anti-stigma interventions, findings 
are reported by intervention type and health care provider 
group. 

Interpretability 

The interpretability of a scale provides information about 
what kind of change in score is needed for results to be 
clinically meaningful [33]. Various measures can assist in 
interpreting scores, including a comparison of means and 
standard deviation (SD) for the OMS-HC among different 
groups. Means are reported for health care providers in 
general, health care provider subtypes, pre/post interven- 
tion scores, and subgroup scores (general population of 
health care providers, by gender and by age groups). Sub- 
group comparisons of means were performed using paired 
t-tests and one-way ANOVA. 
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Results 

Description of the sample population 

Demographic characteristics of the respondents are sum- 
marized in Table 1. Not all study sites reported the same 
demographic characteristics, therefore the following per- 
centages are reported with corresponding sample sizes (n) 
to show how many participants answered each question. 
At baseline, respondents ranged in age from 18 to 65 years 
(n= 1,027), over 77.8% were women (n = 1,029), 18.7% 
reported that they had been treated for a mental illness 
(n = 1,339), 79.6% knew a close family or friend with a 
mental illness (n = 1,414), and 58.8% had treated a patient 
with a mental illness (n = 655). At baseline, the majority of 
health care providers (n = 1,480) were either physicians 
(MDs) (45.0%), nurses (17.0%), medical students (12.7%), 
or students in allied health programs (14.5%), namely 
occupational therapy and pharmacy. The remainder of 
respondents were social workers, psychologists, or other 
non-medical staff. 

Item responses 

A summary of the distribution of responses to each of 
the OMS-HC items are shown in Additional file 2. Floor 
or ceiling effects are considered to be present if more 
than 15% of respondents achieve the lowest or highest 
possible score, respectively [34]. No floor or ceiling ef- 
fects were present. There were low rates of missingness 
on each item, with only 0.1% to 2.7% of participants not 
providing a response to an item, with the exception of 
Item 5 (n = 1,523). A higher proportion of respondents 
(12.1%) did not answer Item 5 "I would be more inclined 
to seek help for a mental illness if my treating health care 
provider was not associated with my workplace". Missing 
data was treated using listwise deletion whereby only 
complete surveys were included in the analysis. 

Factor analysis 

The Kaiser-Meyer-Olkim (KMO) measure of sampling 
adequacy was 0.841, exceeding the benchmark value of 
0.60 [27]. The Bartlett's Test of Spehericity (x 2 = 3969.47, 
p < 0.001) was statistically significant, supporting the fac- 
torability of the correlation matrix. An exploratory factor 
analysis (EFA) of the original 20-item OMS-HC was then 
performed. The EFA found four items (5, 11, 15, 16) with 
weak item-total correlations (below 0.20) and these items 
were dropped to create 16-item scoring of the scale. An 
additional item "If a person with a mental illness complains 
of physical symptoms I would likely attribute this to their 
mental illness" cross-loaded across all three factors equally 
and was dropped leaving 15 remaining items (see Table 2). 
A principal component analysis (PCA) with orthogonal 
varimax rotation and combined scree plot test and parallel 
analysis yielded a three factor solution that accounted 
for 45.3% of the total variance explained. Parallel analysis 



indicated retention of three subscales: Attitude, Disclosure 
and Help-seeking, and Social Distance. Analyses of the 
15-item measure showed that a fixed 3-factor EFA was 
identified as the most stable solution across different 
samples (MDs, medical students, nurses, and allied health 
care providers) (see Additional file 3). EFA and internal 
consistency of the 15-item measure were calculated using 
data from respondents that answered all 15 OMS-HC 
items (n = 1305). 

Internal consistency 

Internal consistency was acceptable for all versions of the 
OMS scale (a = 0.74 to 0.79) and corresponding subscales 
(a = 0.67 to 0.68), including when stratified by health care 
provider group (see Table 3). The internal consistency of 
each subscale {Attitude, Disclosure and Help-seeking, and 
Social Distance) was above 0.65 with the exception of 
medical students (disclosure, a = 0.61) and social workers 
(a < 0.60). The alpha might be lower for the subscales be- 
cause scales with a small number of items can fall below 
the suggested minimum for alpha of 0.70 [35]. 

The OMS-HC subscales were highly correlated with the 
total scale score (attitude: r = 0.82, disclosure: r = 0.73, so- 
cial distance: r = 0.74) and correlated with each other (atti- 
tude and disclosure: r = 0.36, attitude and social distance: 
r = 0.31, disclosure and social distance: r = 0.45). 

Scale distribution and descriptives 

Assessment of composite score histograms for the three 
subscales and total OMS-HC indicated that all of the 
distributions displayed negative skewness (skewness and 
kurtosis, respectively, total: 0.06, 2.93, Attitude: 0.16, 2.71, 
Disclosure and Help-seeking: 0.13, 2.71, Social Distance: 
0.41, 3.38) with Attitude being the most normally dis- 
tributed. A comparative table of means and standard 
deviations by health care provider groups for each ver- 
sion of the OMS-HC is presented in Additional file 4. 
The total and subscale scores were stratified by sub- 
groups: general health care providers, medical specialty, 
socio-demographic characteristics, and study site. Men 
had slightly higher overall baseline stigma scores than 
women, although these differences were not statistically 
significant (t = 1.7, df= 968, p = 0.09). Respondents who 
indicated they had contact or experience with someone 
with a mental illness had lower overall mean scores than 
those who did not (p < 0.001). The same pattern was 
repeated for the three subscales with one exception: there 
was no significant difference on the Disclosure and Help- 
seeking subscale between health care providers who had 
experience treating a person with a mental illness and 
those who did not (t = 1.9, df= 622, p = 0.054). 

Stigma scores across health care provider groups were 
analyzed using a one-way ANOVA. Examination of mean 
scores and corresponding subscales scores indicated that 
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Table 1 Characteristics of the sample, n = 1,523 



Baseline, n 



Pre/Post 
matched, n (%) 

n = 811 (%) 



A) Anti-stigma Workshop 1 

B) Anti-stigma Workshop 2 

C) Anti-stigma Workshop 3 

D) University HCP Program 1 

E) University HCP Program 2 

F) University HCP Program 3 

G) University HCP Program 4 

H) Skills, Physician 1 

I) Skills, ER Nurse/Physician 

J) Other, Hospital Orientation 

K) Other, Community Rounds (BPD) 

Baseline survey only 

L) Physician, Medical Association 

Age 

18-29 

30-39 

40-49 

50-59 

Over 60 

18-25 

26-44 
Gender 

Male 

Female 
Professional group 

Physician (practicing MD) 

Medical student 

Nurse/nurse student 

Social worker 

Psychologist 

Allied (Occupational Therapy/ 
Pharmacy students) 

Non-medica 

Other 

Medical specialization 

Psychiatrists 

Family 

Rural Physician 
Emergency Rural Physician 
Anesthetists 
Surgeons 



9/ 

47 

112 

49 

63 

131 

27 

63 

181 

128 

471 

n = 1,027 (%) 
417 (40.6) 
170 (16.6) 
139 (13.5) 
109 (10.6) 
17 (1.7) 
111 (10.8) 
64 (6.2) 
n = 1,029 (%) 
228 (22.2) 
801 (77.8) 
n = 1,480 (%) 
666 (45.0) 
188 (12.7) 
252 (17.0) 
45 (3.0) 
24 (1.6) 
214 (14.5) 

1 3 (0.9) 

78 (5.3) 

n = 471 (%) 

79 (16.8) 
106 (22.5) 
1 29 (27.4) 
92 (19.5) 
29 (6.2) 
36 (7.6) 



1 55 (85.6) 
26 (26.8) 
24 (51.1) 
96 (85.7) 
41 (83.7) 
63 (100.0) 
88 (67.2) 
16 (59.3) 
21 (33.3) 
1 68 (92.8) 
113 (88.3) 

n/a 



Table 1 Characteristics of the sample, n = 1,523 

(Continued) 

Do you know a close friend or family n = 1,414 (%) 
member with mental illness? 

Yes 1 1 26 (79.6) 

No 255 (18.0) 

Do not know 27(1.9) 

Have you ever been treated for a n = 1,339 (%) 
mental illness? 

Yes 250 (18.7) 

No 1056(78.9) 

Prefer not to answer 33 (2.5) 

Have you ever treated a person with a n = 655 (%) 
mental illness? 

Yes 385 (58.8) 

No 257 (39.2) 

Do not know 12(1.8) 



there were significant differences between the health care 
provider groups (F(6,212) = 4.72, p = 0.0001). Post-hoc 
Tukey's tests showed that physicians, medical students, 
and nurses had significantly higher (more stigmatizing) 
scores on the Attitude and Disclosure and Help-seeking 
subscales than social workers (p < 0.05). There were no 
significant differences on the Social Distance or Disclosure 
and Help-seeking subscales between any of the broad oc- 
cupational groups; however, the available data on medical 
specialists other than psychiatrists (n = 471) revealed that 
in comparison to other specialties (psychiatrists, family 
physicians, and rural physicians), anesthetists and sur- 
geons had significantly higher stigmatizing scores on most 
of the scales (p < 0.01). In addition, scores for family and 
rural physicians were similar (p = 0.6 to 1.0) whereas 
scores for psychiatrists were significantly lower on every 
measure compared to any other medical specialty (p < 
0.05). 



Response rate 

The response rate to the 15-item OMS-HC post-test sur- 
vey was 56.7% (821/1448) and the response rate did not 
differ by gender or age (see Additional file 5). Among 
HCPs, the highest drop in response rate was by physicians: 
45.7% of HCPs who completed the pre-test survey were 
physicians whereas only 22% of HCPs completing both 
the pre-test and post-test surveys were physicians. The 
baseline 15-item OMS-HC mean score did not differ 
between respondents who completed only the pre-test 
(n = 1448, OMS-HC = 2.23, 2.20-2.25) and respondents 
who completed both pre- and post-tests (n = 803, OMS- 
HC = 2.20, 2.18-2.32). Respondents who did not answer 
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Table 2 Factor structure in the 15-item version, n = 1305 







Factor 1 




Item-total 


Alpha if 




1 


2 


3 


correlation 


deleted 


Factor 1: Attitudes of health care providers towards people with mental illness 












am more comfortable helping a person who has a physical illness than 1 am helping a person 

u/hn hac a monta i note (1 nf 
VVMU MdS d llltrMLdl IIIIICSS. Ol ZU^ 


0.6057 


0.2379 


-0.0771 


0.3689 


0.7787 


Despite my professional beliefs, I have negative reactions towards people who have mental 
llness. ( l z ot zU; 


0.6082 


0.1945 


0.1217 


0.4397 


0.7694 


There is little I can do to help peope with mental illness. (13 of 20) 


A "71 1 1 

U./ 1 1 1 


n n373 

U.U3/J 


n 1 i&i 

U. 1 DO/ 


n A A 3 3 


n 77n/i 
U./ /U4 


More than half of people with mental i Iness don't try hard enough to get better. (14 of 20) 


U.D 1 UO 


U.U4Z I 


U.ZO/ D 


n 3n^7 


U./ /4j 


Health care providers do not need to be advocates for peop e with mental illness. (18 of 20) 


0.4538 


— U. I UUU 




0.3214 


0.7793 


struggle to feel compassion for a person with a mental illness. (20 of 20) 


0.61 00 


0.0573 


0.255 1 


r\ A A A n 


0.7702 


Factor 2: Disclosure/help-seeking 












If I were under treatment for a mental illness I would not disclose this to any of my colleagues. 
(Item 4 of 20) 


-0.0294 


0.7514 


0.1150 


0.3252 


0.7805 


I would see myself as weak if I had a mental illness and could not fix it myself. (6 of 20) 


0.3536 


0.6001 


0.0312 


0.4381 


0.7699 


wou d be reluctant to seek he p if I had a mental i ness. (7 of 20) 


0.2961 


0.6152 


0.0270 


0.41 75 


0.7721 


If I had a mental illness, I would tell my friends. (1 0 of 20) 2 


-0.0015 


0.7486 


0.1528 


0.3769 


0.7758 


Factor 3: Social Distance 












If a colleague with whom I work told me they had a managed mental il ness, I would be as 
willing to work with him/her. (3 of 20) 2 


0.0492 


0.0591 


0.6195 


0.2995 


0.7809 


Employers should hire a person with a managed mental illness if he/she is the best person for 
the job. (8 of 20) 2 


0.0762 


0.0612 


0.6886 


0.3569 


0.7774 


would still go to a physician if I knew that the physician had been treated for a mental illness. 
(9 of 20) 2 


0.1159 


0.1536 


0.7083 


0.4244 


0.7725 


would not want a person with a mental illness, even if it were appropriately managed, to work 
with children. (17 of 20) 


0.2777 


0.2553 


0.4954 


0.4618 


0.7695 


would not mind if a person with a mental illness lived next door to me. (1 9 of 20) 2 


0.2047 


0.1176 


0.5881 


0.4044 


0.7736 



1 rotated factor loading using orthogonal varimax from principal component factor (pcf); 2 reverse scored. 
Note: Bartlett's Test of Spehericity (x 2 = 3969.47, p < 0.001) and a Kaiser-Meyer-Olkin was 0.841; alpha = 0.79 
Factor 3); Total variance explained 45.3% (16.2% Factor 1, 15.2% Factor 2, and 13.9% Factor 3); Eigenvalues 
figures show highest factor loading. 



(0.68 for Factor 1, 0.67 for Factor 2, and 0.68 for 
for the three factors were 2.44, 2.28, and 2.08; »bold 



all 15 items (n = 75, mean OMS-HC = 2.33 2.23-2.43) 
had slightly higher mean baseline scores compared to 
those who answered all 15 questions (n = 1448, mean 
OMS-HC = 2.23, 2.20-2.50); however, the difference was 
not statistically significant (f = 1.9, df= 1521, p = Q.Q6). 
The percentage of survey non-completion in the 40 to 



49 age group (12.0%) and 30 to 39 age group (10.0%) 
was higher than in respondents under 30 years of 
age (3.4%). Men (4.0%) had a lower percentage of 
non-completion compared to women (6.2%) although 
the difference was not statistically significant (z = 1.3, 
p = 0.2). 



Table 3 Internal consistency: Cronbach's alpha (a) for each version of the OMS-HC and subscales, stratified by HCP 
group, n = 1305 





n 


20-item version 


12-item version 


15-item version 


Attitude 
(6 items) 


Disclosure/Help-seeking 
(4 items) 


Social Distance 
(5 items) 


Crude (everyone) 


1305 


0.79 


0.74 


0.79 


0.68 


0.67 


0.68 


MD 


830 


0.77 


0.73 


0.77 


0.67 


0.67 


0.66 


Practicing MD 


648 


0.78 


0.73 


0.78 


0.67 


0.69 


0.68 


Medical student 


182 


0.77 


0.74 


0.75 


0.67 


0.61 


0.68 


Nurses 


102 


0.80 


0.74 


0.80 


0.69 


0.66 


0.77 


Allied Health 


214 


0.82 


0.74 


0.81 


0.65 


0.70 


0.69 


Social worker 


35 


0.76 


0.74 


0.81 


0.65 


0.72 


0.57 



•Bold figures, a > 0.70. 
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Responsiveness to change: change scores, effect size and 
standardized response mean (SRM) 

To examine the responsiveness to change of the OMS- 
HC, analyses were conducted using the 15-item version by 
examining the attitudes of participants before (Timei) and 
after (Time 2 ) anti-stigma interventions. Scores for over 
800 complete sets of pre- and post-test questionnaires 
were available. A comparison of pre- and post-test scores 
revealed a significant decrease of 6.6% in the overall mean 
scale score and a decrease of 7.9%, 7.1%, and 4.2% respect- 
ively for the Attitude, Disclosure and Help-seeking, and 
Social Distance subscales after the anti-stigma interven- 
tion (see Additional file 6). The effect size and SRM asso- 
ciated with anti-stigma interventions for the 3-factor 
solution of the OMS-HC scale are provided in Table 4. 
The confidence intervals corresponding to the SRM did 
not contain zero, indicating that the OMS-HC scale 
showed some responsiveness to change. Similar levels 
of change were observed on each of the subscales with 
the exception of Social Distance, which showed the 
smallest level of change. 

If valid, the OMS-HC should reflect change following 
successful anti-stigma interventions, but should show less 
change in samples receiving a small dose of intervention. 
Therefore, it was expected that OMS-HC subscale 
scores in the groups receiving longer (full day to several 
weeks) anti-stigma interventions incorporating contact- 
based educational components [36-38] would increase 
while those receiving shorter (one hour to half-day) in- 
terventions with limited contact-based education [37,38] 
would vary only minimally from a pre- and post-test. Such 
a finding would provide some evidence of construct valid- 
ity for the instrument. OMS-HC post- test data were gath- 
ered after 12 different anti-stigma interventions of different 
lengths. For purposes of comparison, SRM were calculated 
for the respondents who had a total of two administrations 
of the OMS-HC, the second after receiving an anti-stigma 
intervention. As expected, the SRM scores of groups 
receiving longer anti-stigma interventions had larger SRM 
scores compared to groups receiving short anti-stigma 
interventions. For example, the smallest changes were 
observed in shorter courses, often half-day lectures (Skills, 
ER Nurse/Physician, University HCP Program 2) whereas 
courses longer in duration had the highest level of change 
(University HCP Program 4) (see Additional file 7). That 

Table 4 Responsiveness to change, effect size and 
standardized response mean (SRM), n = 803 



Effect size (Cohen's d) SRM (95% CI) 



15- item total 


0.28 (0.04-0.60) 


0.40 (0.33-0.47) 


Attitude 


0.25 (0.10-0.40 ) 


0.32 (0.25-0.39) 


Disdosure/Help-seeking 


0.27 (0.13-0.40 ) 


0.36 (0.33-0.43) 


Social Distance 


0.11 (0.01- 0.23) 


0.14 (0.07-0.21) 



said, research has found many other program elements 
that lead to stigma reductions, such as contact with people 
with mental illness [39-42]. This tentative analyses does 
suggest program length may be another important element 
for programs to consider. 

Discussion 

This study revisited the psychometric properties and factor 
structure of the OMS-HC scale and sought to determine 
the effectiveness of the scale in measuring stigmatizing atti- 
tudes among a variety of health care providers. Findings 
from the current study, using a large sample of health care 
providers, confirmed the satisfactory internal consistency 
of the 12-item and 20-item measure. An exploratory factor 
analysis of this larger sample led to the extraction of a 
3-factor solution of the original 20 OMS-HC items, result- 
ing in subscales of Attitude (6 items), Disclosure and 
Help-seeking (4 items), and Social Distance (5 items). 
Further analyses showed that a fixed 3-factor solution was 
the most stable solution across different samples (MDs, 
medical students, nurses, and allied health care providers), 
indicative of good construct validity. All versions of the 
scale measures and subscales had satisfactory internal 
consistencies (a = 0.67 to 0.79), although there is some 
room for improvement since 0.70 is considered to be 
adequate internal consistency [35]. 

Although the internal consistency of the 20-item scale 
was satisfactory, factor analysis of the 20-item measure re- 
vealed five items that could potentially be dropped. The 
20-item OMS-HC has been successful in detecting posi- 
tive changes during anti-stigma interventions [36,43-47]. 
As such, it remains an acceptable measure for capturing 
health care provider stigma. The current analyses do 
suggest, however, that the 15-item version of the scale is 
superior to the full 20-item. When stratified by health 
care provider group, items loaded onto the three factors. 
With exception of social workers (a < 0.60), the internal 
consistency of the OMS-HC subscales were consistent 
across health care providers (a = 0.61 to 0.81). It is possible 
that items on the scale were capturing concepts that are 
less applicable to social workers. There was a higher pro- 
portion of social workers (21.3%) in the original study 
compared to the current study (3.3%). 

Social workers also had the lowest levels of responsive- 
ness (SRM = 0.05 to 0.21) after receiving anti-stigma train- 
ing. There are several possible reasons for this: social 
workers already had the lowest baseline stigma scores, the 
interventions were not targeted or effective in changing 
stigmatizing attitudes of social workers, or items on the 
scale were not specific enough to capture changes in atti- 
tudes of social workers. Further analysis of scale respon- 
siveness would suggest that some anti-stigma interventions 
are better at addressing some aspects of stigma related to 
mental illness compared to others. For example, a primary 
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focus of program H (Skills, Physician 1) was on reducing 
social distance. Compared to the other interventions, this 
program showed the highest level of responsiveness on the 
Social Distance subscale (SRM = 0.50, -0.06-1.06). 

The moderate associations found between the three 
factors (Attitude, Disclosure and Help-seeking, and Social 
Distance) support the notion of a shared conceptual 
theme. Barney et al. [48] suggest that the convergent 
association between social distance and disclosure and 
help-seeking is consistent with the notion that people 
with a mental illness who believe they are flawed and 
unworthy would view others the same way. It is possible 
that among health care providers, self-stigma arises 
from the awareness of, and agreement with, negative 
stereotypical beliefs [6] held by their peers. 

Several approaches have now been developed and are 
beginning to be evaluated for combating mental illness 
stigma among health care providers [36,43,44,46,47]. An 
important question is whether the OMS-HC is sensitive 
to changes achieved by various anti-stigma interventions. 
In this study, the OMS-HC has shown moderate sensi- 
tivity to change before and after various types of anti- 
stigma interventions among different groups of health 
care providers. These findings are compelling because 
the sensitivity to change occurred across multiple sam- 
ples, including university-based interventions of medical, 
pharmacy, and occupational therapy students, as well as 
in anti-stigma workshops offered to practicing health 
care providers. In this study, the items about potential 
behaviour on the Social Distance subscale had the weakest 
level of responsiveness (SRM < 0.50) whereas the more 
attitudinal-based items of the Attitude (SRM < 0.91) and 
Disclosure and Help-seeking (SRM < 0.68) subscales had 
stronger responsiveness. Most of the programs included in 
this study tended to focus more on improving attitudes 
and perceptions of mental illness/persons with mental ill- 
ness than on reducing social distance per se. In a meta- 
analysis of outcome studies, researchers [39] found that 
anti-stigma interventions had more impact on attitudes 
than behaviour intentions and this may partially account 
for the lower levels of responsiveness on the Social 
Distance subscale in our findings. 

Comparisons among the subscale scores on the OMS- 
HC can provide an indication of which aspects of stigma 
an intervention was most impactful at addressing. For ex- 
ample, an anti-stigma program can target simultaneously 
attitudes towards help-seeking and social distance. In an 
evaluation of this hypothetical program, more movement 
from pre to post on the help-seeking subscale than the 
social distance subscale would be an indicator that the 
program was more effective at addressing the former 
aspect of stigma than the latter. Notably in the current 
study, the single anti-stigma program which most em- 
phasized concepts related to Social Distance showed the 



greatest change (SRM = 0.50, -0.06-1.06) on this con- 
struct in comparison to anti-stigma interventions which 
did not particularly focus on this topic (SRM = 0.01 to 
0.30). Additionally, this finding is promising because it 
demonstrates that although there was less movement 
on this subscale in general for the entire sample, it is 
possible to measure reductions in social distance. In 
stigma research, social distance has been identified as a 
proxy measure of discriminating behaviour [49] . There- 
fore, it is possible that movement on this particular sub- 
scale could be demonstrating that anti-stigma programs 
can change discriminating behaviour. 

The strengths of this study include the use of EFA 
methods and assessment of sensitivity to change in a 
large representative sample of health care providers to 
which anti-stigma programs are targeted. Overall, the 15- 
item version displayed acceptable internal consistency, 
construct validity, responsiveness to change, and a mean- 
ingful factorial structure. However, some limitations of 
this study must be taken into account. First, sociodemo- 
graphic information was not available for the entire study 
sample as this information was limited to what was pro- 
vided by each of the anti-stigma programs. Therefore, 
caution should be used when interpreting the results 
tied to this type of information. Second, it appears that 
in a small number of cases, matching of pre and post 
surveys was unsuccessful because respondents were asked 
to create and recall their original study ID number. Third, 
as the OMS-HC is a self-report measure of stigma, it may 
underestimate the level of an individual's personal stigma. 
Researchers [50] have postulated that social desirability 
bias may operate in self-report measures and cannot be 
excluded from the OMS-HC or any other stigma scale 
[51] based on self-report. However, we can be confident 
that the representative sample in this study will provide a 
reliable indication of the minimum prevalence of stigma 
in health care providers. Another potential limitation is 
that, even though scales such as the OMS-HC offer a rea- 
sonable indication of the beliefs that support behavioural 
attitudes, they have limitations in capturing actual behav- 
ioural responses. Studies using other attitudinal, behav- 
ioural, or implicit measures of stigma related to mental 
illness may shed further light on the validity of the 
OMS-HC measure. Ideally, future evaluations of anti- 
stigma interventions targeting health care providers 
should measure the impact and health outcomes of pa- 
tients to determine the full trickle-down effects of those 
interventions. This type of research could also address 
strategies to measure behavioural and economic out- 
comes, including the impact on the utilization of health 
care services. Other future research could explore why 
certain OMS-HC item were more likely to be unanswered 
and to determine the best techniques to deal with item 
non-response and missing data. Finally, the OMS-HC 
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could be translated into other languages and testing be 
carried out to determine its reliability and validity across 
different cultures and parts of the world. 

Conclusions 

Research has suggested that implementing changes in 
training and continuing education for health care providers 
to reduce stigmatizing attitudes toward mental illness 
have the potential to positively influence clinical prac- 
tice [37,52-56] and instruments such as the OMS-HC 
can be used to accurately and reliably measure the ef- 
fectiveness of such programs. Our results suggest that 
the 15-item OMS-HC scale appears to be superior to 
the 12-item version. Overall, our results support the use 
of the 15-item scale, with the calculation of three sub 
scales for Attitude, Disclosure and Help-seeking, and 
Social Distance. Future investigators should consider 
using the OMS-HC to examine the stigma in health 
care providers and as a tool to measure effectiveness of 
anti-stigma programs in such populations. 
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